We study finite-sum nonconvex optimization problems, where the objective function is an average of $n$ nonconvex functions. We propose a new stochastic gradient descent algorithm based on nested variance reduction. Compared with conventional stochastic variance reduced gradient (SVRG) algorithm that uses two reference points to construct a semi-stochastic gradient with diminishing variance in each iteration, our algorithm uses $K+1$ nested reference points to build a semi-stochastic gradient to further reduce its variance in each iteration.

Multilevel Monte-Carlo (MLMC) technique and establish its sample and computational complexities.

In this paper we study the second-order optimality of decentralized stochastic algorithm that escapes saddle point efficiently for nonconvex optimization problems.

Then momentum acceleration is also introduced into our dual accelerating update step.

Then momentum acceleration is also introduced into our dual accelerating update step.

Despite its success in practice, theoretical analyses of Adam are still limited.

This is meaningful because in most applications, the time complexities for evaluating gradients at different points are of the same magnitude.

Neural Information Processing Systems http://nips.cc/